Material capture using imaging

ABSTRACT

Methods and systems are provided for performing material capture to determine properties of an imaged surface. A plurality of images can be received depicting a material surface. The plurality of images can be calibrated to align corresponding pixels of the images and determine reflectance information for at least a portion of the aligned pixels. After calibration, a set of reference materials from a material library can be selected using the calibrated images. The set of reference materials can be used to determine a material model that accurately represents properties of the material surface.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to, U.S.patent application Ser. No. 15/589,757, filed on May 8, 2017, entitledMATERIAL CAPTURE USING IMAGING, the entirety of which is incorporatedherein by reference.

BACKGROUND

Oftentimes, users desire to use the appearance of real-world objects asreferences for the surfaces of computer-generated objects; for example,using the appearance of a wood-grain desk in their workspace as areference for the surface of a computer-generated desk. Creating a modelof surface properties can allow the appearance of real-world objects tobe applied in such a manner. Determining a bi-directional reflectancedistribution function (BRDF) that defines how light is reflected at asurface can be used to create such a material model. Finding a BRDF of asurface allows for capture of the appearance of real-world surfaces foruse as references when rendering computer-generated objects to producehigh-quality photorealistic content. As such, the process of determininga material model that can accurately assign the appearance of materialsurface to a computer-generated object can be known as BRDF capture.

Currently, BRDF capture can be performed using imaging to determineinformation about the appearance of a surface of a real-world object.Typically, to perform BRDF capture, existing techniques rely on complexsetups to collect photographs under a multitude of staged lightingconditions. The requirements of such a set-up make BRDF captureimpracticable and inefficient as use of these complex systems is highlycostly and requires significant time and effort. Approaches that attemptto overcome such complex system setups are limited to BRDF capture ofhighly uniform material surfaces where there are little to no variationsin the surface properties across the material surface. As such, theseapproaches fail to allow for accurate BRDF capture of surfaces withmaterial variations, for example, differences in wood grain texture,etchings in a metal surface, or variations in leather coloration acrossa captured surface.

SUMMARY

Embodiments of the present invention are directed towards enablingaccurate material capture of the properties of a surface. In accordancewith embodiments of the present invention, material capture uses imagingto create a model of material properties of a surface that can be usedto assign the properties to the surface of a computer-generated object.To generate an accurate material model, material capture takes images ofa surface in a real-world setting. Once gathered, this information canbe used to build a material model that estimates the reflectanceproperties of the material surface to accurately reproduce the visualappearance of the material surface using computer graphics.

Creating such a material model can be accomplished by capturing imagesof a real-world surface. These images can then be calibrated by aligningthe pixels of the images to compile the different lighting/cameraposition information available from each captured image at each pixel.This lighting/camera position information can be used to select a set ofreference materials that can be used to extrapolate the materialproperties of the imaged material surface. Such a library of referencedmaterials can be narrowed to a set that represents the properties of theimaged material surface. This subset of reference materials can then beused to estimate the properties of the imaged material surface based onpreviously determined material properties of the reference materials.Specifically, the properties of the imaged material surface can bedetermined by iteratively using the previously determined materialproperties of the reference materials until a material model is producedthat is representative of the properties of the imaged material surface.Such a material model can then be used to reproduce the visualappearance of the imaged material surface for surfaces ofcomputer-generated objects.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a diagram of an environment in which one or moreembodiments of the present disclosure can be practiced, in accordancewith various embodiments of the present disclosure.

FIG. 2 depicts an example configuration of an operating environment inwhich some implementations of the present disclosure can be employed, inaccordance with various embodiments of the present disclosure.

FIG. 3 depicts aspects of an illustrative material capture system, inaccordance with various embodiments of the present disclosure.

FIG. 4 depicts aspects of an illustrative refining engine, in accordancewith various embodiments of the present disclosure.

FIG. 5 illustrates a process flow depicting an embodiment performingmaterial capture for a surface material, in accordance with variousembodiments of the present disclosure.

FIG. 6 illustrates a process flow depicting an example calibration ofreceived captured images of a material surface during material capture,in accordance with various embodiments of the present disclosure.

FIG. 7 is a block diagram of an example computing device in whichembodiments of the present disclosure may be employed.

DETAILED DESCRIPTION

When rendering computer-generated objects, users oftentimes want toutilize the appearance of real-world objects to guide the appearanceassigned to the surfaces of objects during rendering. For example, auser may wish to create a computer-generated image of a desk thatimitates the appearance of a wood-grain desk in their workspace.Accurate determination of the properties of the material(s) thatcomprise the surface of a real-world object enables the creation of amaterial model of the surface material of the object. Such a materialmodel can then be applied when creating computer-generated objects.Creating material models can be performed using a bi-directionalreflectance distribution function (BRDF). A BRDF is a function thatdefines how light is reflected at a surface. The function takes incominglight direction and outgoing light direction to determine the ratio ofreflected radiance to the irradiance incident on the surface.Determining a BRDF can aid in capturing of the appearance of real-worldsurfaces as a material model to use as a reference when rendering acomputer-generated object to produce high-quality photorealisticcontent. As such, the process of determining a material model that canaccurately assign the appearance of material surface to acomputer-generated object can be known as material capture.

Generally, in material capture, the reflectance of a material(s), whichcan be represented as a BRDF, that comprises a surface of an object issampled using a variety of lighting-view combinations. Once gathered,this reflectance information can be used to build a material model thatestimates the properties of the surface to accurately reproduce theappearance of the surface using computer graphics. As such, a materialmodel determined using material capture can be used in computer graphicsto assign information taken from a real-world surface, such as color,patterning, and/or lighting, to a surface of an computer-generatedobject during rendering.

Existing methods to determine a BRDF of a material surface utilizecomplex acquisition systems to collect surface material information.Such complex systems make material capture impracticable andinefficient, as use of these systems is costly in time, money, andeffort. While approaches have been introduced that attempt to overcomesuch limitations by minimizing the complexity of the setup, suchapproaches are limited to determining a BRDF for stationary textures.Stationary textures belong to material surfaces with a large degree ofredundancy in that there are multiple points on the material surfacethat exhibit identical, or highly similar, surface properties. As such,these approaches use this similarity to combine information frommultiple places on the surface to determine surface properties of thematerial surface as a whole. However, using this approach and relying onsimilarities across a surface result in the assumption that the surfacehas the same properties at points (e.g., all points) across a sample.Such an assumption prevents accurate capture of surfaces which havedistinct textures in different regions.

For example, one conventional method employs the use of two images, onetaken with a flash and one taken without a flash, where the flash imageprovides lighting-dependent samples of the material surface and thenon-flash image allows for identification of similar areas across thesurface. These images can then be used to determine various areas acrossthe flash image for which to gather the information necessary todetermine the BRDF for the surface. As such, reflectance information isaugmented using multiple areas of a surface to synthesize reflectancesamples of the surface. Because only one measurement of reflectanceinformation per pixel exists, such a method is not capable ofdetermining a BRDF at each pixel on the surface without using theaugmented information. However, when such augmented information is used,the determined BRDF no longer accurately captures variations in thesurface and instead finds an average BRDF across the surface as a whole.As such, this method only works for stationary textures and cannothandle surfaces with distinct patterns or other such surface variations.Thus, such approaches fail to allow for accurate determination of amodel indicating variations in BRDF across surfaces due to surfaceand/or material property variations, such as, for example, differencesin wood grain texture, differences in etchings in a metal surface, orvariations in leather coloration.

As such, embodiments of the present invention are directed towardsfacilitating efficient and effective modeling of material properties,such as reflectance, of a surface using material capture. Specifically,material capture uses a simplistic setup while still allowing accuratecapture of non-stationary textures, such as, determining accuratematerial models for material surfaces with irregularities,dissimilarities, or other such variations across the surfaces. Such anaccurate material model is possible because multiple measurements ofreflectance information are collected at individual pixels across thesurface which can be used to determine variations in BRDF acrosssurface. In this way, embodiments of the present invention can allow fordetermining a per-pixel BRDF of a material surface.

At a high level, in accordance with embodiments of the presentinvention, to perform such material capture, multiple images of amaterial surface can be taken and aligned without user intervention todetermine lighting/view directions, compiled from images atcorresponding pixels. In this way, images can be taken of a real-worldsurface using any device with image capturing capabilities, such as, forexample, a mobile phone. These images can then be calibrated by aligningthe images and determining the different lighting/camera positioninformation available from each image at corresponding pixels across theimaged surface. Such calibrated images can be used to determine the BRDFand shape of the object surface, where the shape can be characterized bysurface normals. A surface normal is a vector that is perpendicular tothe tangent plane at a point, or pixel, on the surface.

When determining material properties of a surface during materialcapture, reflectance information about a surface material can beestimated using a database, such as a material library, of referencematerials that have had such information previously calculated. As such,the BRDF at a pixel can be represented using a subset of referencematerials selected from a dictionary of reference materials. The BRDF ata pixel can be expressed using the various reference materials used todetermine an accurate material model for the properties of the surfacematerial. Advantageously, using a compilation of such referencematerials to represent the properties of the surface material greatlyreduces the number of unknowns that must be determined to find the BRDF.Specifically, instead of having to solving for a high-dimensionalvector, only percentages of contribution from the reference materialsneeds to be solved for each pixel.

To determine the BRDF at a pixel using such reference materials,iterative updates can be performed until the error in the output isreduced to below a predetermined threshold. To perform thisdetermination, given the reference materials, and the coefficient ofabundance based on the subset of reference materials, the surface normalcan be solved for by searching for estimates of the surface normals thatminimize the energy function. Upon finding new surface normals, the newsurface normals can then be used to solve for new abundances. Thisprocess can be iteratively performed until the output falls below apredetermined threshold level.

Turning to FIG. 1, FIG. 1 is a diagram of an environment 100 in whichone or more embodiments of the present disclosure can be practiced. Theenvironment 100 includes one or more user devices, such as user devices102A-102N. Examples of a user device include, but are not limited to, apersonal computer (PC), a tablet computer, a desktop computer, aprocessing unit, any combination of these devices, or any other suitabledevice having one or more processors. Each user device can include atleast one application supported by the creative apparatus 108. It is tobe appreciated that following description may generally refer to theuser device 102A as an example and any other user device can be used.

A user of the user device can utilize various products, applications, orservices supported by the creative apparatus 108 via the network 106.The user devices 102A-102N can be operated by various users. Examples ofthe users include, but are not limited to, creative professionals orhobbyists who use creative tools to generate, edit, track, or managecreative content, advertisers, publishers, developers, content owners,content managers, content creators, content viewers, content consumers,designers, editors, any combination of these users, or any other userwho uses digital tools to create, edit, track, or manages digitalexperiences.

A digital tool, as described herein, includes a tool that is used forperforming a function or a workflow electronically. Examples of adigital tool include, but are not limited to, content creation tool,content editing tool, content publishing tool, content tracking tool,content managing tool, content printing tool, content consumption tool,any combination of these tools, or any other tool that can be used forcreating, editing, managing, generating, tracking, consuming orperforming any other function or workflow related to content. A digitaltool includes the creative apparatus 108.

Digital experience, as described herein, includes experience that can beconsumed through an electronic device. Examples of the digitalexperience include content creating, content editing, content tracking,content publishing, content posting, content printing, content managing,content viewing, content consuming, any combination of theseexperiences, or any other workflow or function that can be performedrelated to content.

Content, as described herein, includes electronic content. Examples ofthe content include, but are not limited to, image, video, website,webpage, user interface, menu item, tool menu, magazine, slideshow,animation, social post, comment, blog, data feed, audio, advertisement,vector graphic, bitmap, document, any combination of one or morecontent, or any other electronic content.

User devices 102A-102N can be connected to a creative apparatus 108 viaa network 106. Examples of the network 106 include, but are not limitedto, internet, local area network (LAN), wireless area network, wiredarea network, wide area network, and the like.

The creative apparatus 108 includes one or more engines for providingone or more digital experiences to the user. The creative apparatus 108can be implemented using one or more servers, one or more platforms withcorresponding application programming interfaces, cloud infrastructureand the like. In addition, each engine can also be implemented using oneor more servers, one or more platforms with corresponding applicationprogramming interfaces, cloud infrastructure and the like. The creativeapparatus 108 also includes a data storage unit 112. The data storageunit 112 can be implemented as one or more databases or one or more dataservers. The data storage unit 112 includes data that is used by theengines of the creative apparatus 108.

A user of the user device 102A visits a webpage or an application storeto explore applications supported by the creative apparatus 108. Thecreative apparatus 108 provides the applications as a software as aservice (SaaS), or as a standalone application that can be installed onthe user device 102A, or as a combination. The user can create anaccount with the creative apparatus 108 by providing user details andalso by creating login details. Alternatively, the creative apparatus108 can automatically create login details for the user in response toreceipt of the user details. In some embodiments, the user is alsoprompted to install an application manager. The application managerenables the user to manage installation of various applicationssupported by the creative apparatus 108 and also to manage otherfunctionalities, such as updates, subscription account and the like,associated with the applications. The user details are received by auser management engine 116 and stored as user data 118 in the datastorage unit 112. In some embodiments, the user data 118 furtherincludes account data 120 under which the user details are stored.

The user can either opt for a trial account or can make payment based ontype of account or subscription chosen by the user. Alternatively, thepayment can be based on product or number of products chosen by theuser. Based on payment details of the user, a user operational profile122 is generated by an entitlement engine 124. The user operationalprofile 122 is stored in the data storage unit 112 and indicatesentitlement of the user to various products or services. The useroperational profile 122 also indicates type of user, i.e. free, trial,student, discounted, or paid.

In some embodiment, the user management engine 116 and the entitlementengine 124 can be one single engine performing the functionalities ofboth the engines.

The user can then install various applications supported by the creativeapparatus 108 via an application download management engine 126.Application installers or application programs 128 present in the datastorage unit 112 are fetched by the application download managementengine 126 and made available to the user directly or via theapplication manager. In one embodiment, an indication of all applicationprograms 128 are fetched and provided to the user via an interface ofthe application manager. In another embodiment, an indication ofapplication programs 128 for which the user is eligible based on user'soperational profile are displayed to the user. The user then selects theapplication programs 128 or the applications that the user wants todownload. The application programs 128 are then downloaded on the userdevice 102A by the application manager via the application downloadmanagement engine 126. Corresponding data regarding the download is alsoupdated in the user operational profile 122. An application program 128is an example of the digital tool. The application download managementengine 126 also manages the process of providing updates to the userdevice 102A.

Upon download, installation and launching of an application program, inone embodiment, the user is asked to provide the login details. A checkis again made by the user management engine 116 and the entitlementengine 124 to ensure that the user is entitled to use the applicationprogram. In another embodiment, direct access is provided to theapplication program as the user is already logged into the applicationmanager.

The user uses one or more application programs 104A-104N installed onthe user device to create one or more projects or assets. In addition,the user also has a workspace within each application program. Theworkspace, as described herein, includes setting of the applicationprogram, setting of tools or setting of user interface provided by theapplication program, and any other setting or properties specific to theapplication program. Each user can have a workspace. The workspace, theprojects, and/or the assets can be stored as application program data130 in the data storage unit 112 by a synchronization engine 132.Alternatively or additionally, such data can be stored at the userdevice, such as user device 102A.

The application program data 130 includes one or more assets 140. Theassets 140 can be a shared asset which the user wants to share withother users or which the user wants to offer on a marketplace. Theassets 140 can also be shared across multiple application programs 128.Each asset includes metadata 142. Examples of the metadata 142 include,but are not limited to, font, color, size, shape, coordinate, acombination of any of these, and the like. In addition, in oneembodiment, each asset also includes a file. Examples of the fileinclude, but are not limited to, an image 144, text 146, a video 148, afont 150, a document 152, a combination of any of these, and the like.In another embodiment, an asset only includes the metadata 142.

The application program data 130 also include project data 154 andworkspace data 156. In one embodiment, the project data 154 includes theassets 140. In another embodiment, the assets 140 are standalone assets.Similarly, the workspace data 156 can be part of the project data 154 inone embodiment while it may be standalone data in other embodiment.

A user can operate one or more user devices to access data. In thisregard, the application program data 130 is accessible by a user fromany device, including a device which was not used to create the assets140. This is achieved by the synchronization engine 132 that stores theapplication program data 130 in the data storage unit 112 and enablesthe application program data 130 to be available for access by the useror other users via any device. Before accessing the application programdata 130 by the user from any other device or by any other user, theuser or the other user may need to provide login details forauthentication if not already logged in. In some cases, if the user orthe other user are logged in, then a newly created asset or updates tothe application program data 130 are provided in real time. The rightsmanagement engine 136 is also called to determine whether the newlycreated asset or the updates can be provided to the other user or not.The workspace data 156 enables the synchronization engine 132 to providea same workspace configuration to the user on any other device or to theother user based on the rights management data 138.

In various embodiments, various types of synchronization can beachieved. For example, the user can pick a font or a color from the userdevice 102A using a first application program and can use the font orthe color in a second application program on any other device. If theuser shares the font or the color with other users, then the other userscan also use the font or the color. Such synchronization generallyhappens in real time. Similarly, synchronization of any type of theapplication program data 130 can be performed.

In some embodiments, user interaction with the applications 104 istracked by an application analytics engine 158 and stored as applicationanalytics data 160. The application analytics data 160 includes, forexample, usage of a tool, usage of a feature, usage of a workflow, usageof the assets 140, and the like. The application analytics data 160 caninclude the usage data on a per user basis and can also include theusage data on a per tool basis or per feature basis or per workflowbasis or any other basis. The application analytics engine 158 embeds apiece of code in the applications 104 that enables the application tocollect the usage data and send it to the application analytics engine158. The application analytics engine 158 stores the usage data as theapplication analytics data 160 and processes the application analyticsdata 160 to draw meaningful output. For example, the applicationanalytics engine 158 can draw an output that the user uses “Tool 4”maximum number of times. The output of the application analytics engine158 is used by a personalization engine 162 to personalize tool menu forthe user to show “Tool 4” on top. Other types of personalization canalso be performed based on the application analytics data 158. Inaddition, the personalization engine 162 can also use the workspace data156 or the user data 118 including user preferences to personalize oneor more application programs 128 for the user.

In some embodiments, the application analytics data 160 includes dataindicating status of project of the user. For example, if the user waspreparing an article in a digital publishing application and what wasleft was publishing the prepared article at the time the user quit thedigital publishing application then the application analytics engine 158tracks the state. Now when the user next opens the digital publishingapplication on another device then the user is indicated the state andoptions are provided to the user for publishing using the digitalpublishing application or any other application. In addition, whilepreparing the article, a recommendation can also be made by thesynchronization engine 132 to incorporate some of other assets saved bythe user and relevant for the article. Such a recommendation can begenerated using one or more engines, as described herein.

The creative apparatus 108 also includes a community engine 164 whichenables creation of various communities and collaboration among thecommunities. A community, as described herein, includes a group of usersthat share at least one common interest. The community can be closed,i.e., limited to a number of users or can be open, i.e., anyone canparticipate. The community enables the users to share each other's workand comment or like each other's work. The work includes the applicationprogram data 140. The community engine 164 stores any data correspondingto the community, such as work shared on the community and comments orlikes received for the work as community data 166. The community data166 also includes notification data and is used for notifying otherusers by the community engine in case of any activity related to thework or new work being shared. The community engine 164 works inconjunction with the synchronization engine 132 to provide collaborativeworkflows to the user. For example, the user can create an image and canrequest for some expert opinion or expert editing. An expert user canthen either edit the image as per the user liking or can provide expertopinion. The editing and providing of the expert opinion by the expertis enabled using the community engine 164 and the synchronization engine132. In collaborative workflows, a plurality of users are assigneddifferent tasks related to the work.

The creative apparatus 108 also includes a marketplace engine 168 forproviding marketplace to one or more users. The marketplace engine 168enables the user to offer an asset for selling or using. The marketplaceengine 168 has access to the assets 140 that the user wants to offer onthe marketplace. The creative apparatus 108 also includes a searchengine 170 to enable searching of the assets 140 in the marketplace. Thesearch engine 170 is also a part of one or more application programs 128to enable the user to perform search for the assets 140 or any othertype of the application program data 130. The search engine 170 canperform a search for an asset using the metadata 142 or the file.

The creative apparatus 108 also includes a document engine 172 forproviding various document related workflows, including electronic ordigital signature workflows, to the user. The document engine 172 canstore documents as the assets 140 in the data storage unit 112 or canmaintain a separate document repository (not shown in FIG. 1).

In accordance with embodiments of the present invention, applicationprograms 128 can include an application, such as application 210 of FIG.2, which facilitates material capture of an imaged surface. Such anapplication can be provided to the user device 102A so that the materialcapture application operates via the user device. In another embodiment,material capture functionality can be provided as an add-on or plug-into an application, such as a design or image processing application.

FIG. 2 depicts an example configuration of an operating environment inwhich some implementations of the present disclosure can be employed, inaccordance with various embodiments of the present disclosure. It shouldbe understood that this and other arrangements described herein are setforth only as examples. Other arrangements and elements (e.g., machines,interfaces, functions, orders, and groupings of functions, etc.) can beused in addition to or instead of those shown, and some elements may beomitted altogether for the sake of clarity. Further, many of theelements described herein are functional entities that may beimplemented as discrete or distributed components or in conjunction withother components, and in any suitable combination and location. Variousfunctions described herein as being performed by one or more entitiesmay be carried out by hardware, firmware, and/or software. For instance,some functions may be carried out by a processor executing instructionsstored in memory as further described with reference to FIG. 7.

It should be understood that operating environment 200 shown in FIG. 2is an example of one suitable operating environment. Among othercomponents not shown, operating environment 200 includes a number ofuser devices, such as user devices 202 a and 202 b through 202 n,network 204, and server(s) 208. Each of the components shown in FIG. 2may be implemented via any type of computing device, such as one or moreof computing device 700 described in connection to FIG. 7, for example.These components may communicate with each other via network 204, whichmay be wired, wireless, or both. Network 204 can include multiplenetworks, or a network of networks, but is shown in simple form so asnot to obscure aspects of the present disclosure. By way of example,network 204 can include one or more wide area networks (WANs), one ormore local area networks (LANs), one or more public networks such as theInternet, and/or one or more private networks. Where network 204includes a wireless telecommunications network, components such as abase station, a communications tower, or even access points (as well asother components) may provide wireless connectivity. Networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets, and the Internet. Accordingly, network 204 is notdescribed in significant detail.

It should be understood that any number of user devices, servers, andother components may be employed within operating environment 200 withinthe scope of the present disclosure. Each may comprise a single deviceor multiple devices cooperating in a distributed environment.

User devices 202 a through 202 n can be any type of computing devicecapable of being operated by a user. For example, in someimplementations, user devices 202 a through 202 n are the type ofcomputing device described in relation to FIG. 7. By way of example andnot limitation, a user device may be embodied as a personal computer(PC), a laptop computer, a mobile device, a smartphone, a tabletcomputer, a smart watch, a wearable computer, a personal digitalassistant (PDA), an MP3 player, a global positioning system (GPS) ordevice, a video player, a handheld communications device, a gamingdevice or system, an entertainment system, a vehicle computer system, anembedded system controller, a remote control, an appliance, a consumerelectronic device, a workstation, any combination of these delineateddevices, or any other suitable device.

The user devices can include one or more processors, and one or morecomputer-readable media. The computer-readable media may includecomputer-readable instructions executable by the one or more processors.The instructions may be embodied by one or more applications, such asapplication 210 shown in FIG. 2. Application 210 is referred to as asingle application for simplicity, but its functionality can be embodiedby one or more applications in practice. As indicated above, the otheruser devices can include one or more applications similar to application210.

The application(s) may generally be any application capable offacilitating the exchange of information between the user devices andthe server(s) 208 in carrying out material capture for a materialsurface. In some implementations, the application(s) comprises a webapplication, which can run in a web browser, and could be hosted atleast partially on the server-side of environment 200. In addition, orinstead, the application(s) can comprise a dedicated application, suchas an application having image processing functionality. In some cases,the application is integrated into the operating system (e.g., as aservice). It is therefore contemplated herein that “application” beinterpreted broadly.

In accordance with embodiments herein, the application 210 canfacilitate material capture of an imaged material surface. Inparticular, a user can select or input images and/or a video of thesurface of an object. In some embodiments, the surface is generallyplanar with minor variations. A user may select desired images from arepository, for example, stored in a data store accessible by a networkor stored locally at the user device 202 a. Alternatively, a user canselect or input images and/or a video using, for example, a camera on adevice, for example, user device 202 a.

Based on the input images and/or video, material capture of a surfacecan be determined, for instance, at a server, and provided to the userdevice 202 a. In this regard, a material model representing theproperties of the surface material can be accessible to a user, forexample, within an application, allowing the user to apply theattributes of the surface to the surface of a computer-generated object.

As described herein, server 208 can facilitate material capture viamaterial capture system 206. To perform material capture, server 208 caninteract with a user device (e.g., a camera) with imaging functionality.Server 208 includes one or more processors, and one or morecomputer-readable media. The computer-readable media includescomputer-readable instructions executable by the one or more processors.The instructions may optionally implement one or more components ofmaterial capture system 206, described in additional detail below.

For cloud-based implementations, the instructions on server 208 mayimplement one or more components of material capture system 206, andapplication 210 may be utilized by a user to interface with thefunctionality implemented on server(s) 208. In some cases, application210 comprises a web browser. In other cases, server 208 may not berequired. For example, the components of material capture system 206 maybe implemented completely on a user device, such as user device 202 a.In this case, material capture system 206 may be embodied at leastpartially by the instructions corresponding to application 210.

Thus, it should be appreciated that material capture system 206 may beprovided via multiple devices arranged in a distributed environment thatcollectively provide the functionality described herein. Additionally,other components not shown may also be included within the distributedenvironment. In addition, or instead, material capture system 206 can beintegrated, at least partially, into a user device, such as user device202 a. Furthermore, material capture system 206 may at least partiallybe embodied as a cloud computing service.

FIG. 3 depicts an example configuration of an illustrative materialcapture system 304 for implementing material capture of a surface, inaccordance with various embodiments of the present disclosure. Materialcapture system 304 can be any type of processing system executed byinstructions on a computing device. It should be understood that thisand other arrangements described herein are set forth only as examples.Other arrangements and elements (e.g., machines, interfaces, functions,orders, and groupings of functions, etc.) can be used in addition to orinstead of those shown, and some elements may be omitted altogether forthe sake of clarity. Further, many of the elements described herein arefunctional entities that may be implemented as discrete or distributedcomponents or in conjunction with other components, and in any suitablecombination and location. Various functions described herein as beingperformed by one or more entities may be carried out by hardware,firmware, and/or software. For instance, some functions may be carriedout by a processor executing instructions stored in memory as furtherdescribed with reference to FIG. 7. As depicted in FIG. 3, materialcapture system 304 includes capture engine 306, calibration engine 308,and modeling engine 310.

Capture engine 306 obtains references or receives images 302 taken of asurface. Such images can be used to produce a material model of thesurface captured in the images. Images 302 can be taken using a userdevice, for example, user device 202 a-202 n as described with referenceto FIG. 2. Such a user device can be a portable user device coupled withan illuminant, for example, a mobile phone with a flash. Images caninclude two photographs captured at various locations, one image undernormal lighting and another image using additional lighting such as, forexample, a flash. Alternatively, images 202 can be a short video, orsequence of images, of a surface (e.g., planar surface) using, forexample, a mobile phone. In embodiments, images captured using a videocan be taken by moving a camera planar to a surface. For example, acamera can be moved in a pattern or grid over a surface, with anilluminated light source, such as a flash. While four images are shownin FIG. 3, it should be appreciated that any number of images and/or anylength of video sequence can be received by capture engine 306.

After capture engine 306 obtains references or receives images 302,calibration engine 308 generally performs calibration. In particular,images can be calibrated by first aligning pixels in the receivedimages. Such alignment allows for compiling the different lightingand/or camera positions at one or more corresponding pixels of the samelocation on the surface. To aid in alignment, a grid can be used as areference to align the images, for instance, grid points of acheckerboard pattern can be used to determine a rotation and translationmatrix for each camera position. Such camera position differences can beused to align the images so that information at one or more pixels canbe compiled across the images. Such a checkerboard pattern can be placedon the surface being imaged. In other embodiments, other types ofmarkers can be used to align the images, for example, using a designatedreference point.

After aligning pixels, calibration continues by determining the lightingand view direction at pixels using the aligned images. In embodiments,because a light source is rigidly fixed to a camera used to take theimages, the position of the light source can be determined from theposition of the camera using a pre-determined rotation and translation.Further, a light source can be assumed to be of constant brightness, andthe radiometric response of the camera can be assumed to be linear.Radiometric response is a function that transforms sensor irradianceinto measured intensities that are the output from the camera.

Because the relative position between the camera and the light source isknown, the position of the light source can easily be calculated. Assuch, the three-dimensional position for pixels in the images can bedetermined by computing the positions using specified coordinates on,for example, a reference checkerboard. It should be noted that such adetermination typically may not provide accurate surface normalestimations. However, using the reference, the different images can bealigned so that all the lighting information at an individual pixelthroughout the images can be compiled from the images. Such informationfrom corresponding pixels across information can be used to helpdetermine the lighting and view direction information at points on theimaged surface. This lighting and view information can then be used todetermine properties of the surface during material capture to generatea material model.

Once calibration is completed, modeling engine 310 can use thelighting/view information to determine a material model for the materialsurface. This material model can be based on the BRDF of referencematerials, where reference materials are materials exhibiting similarsurface properties as the imaged surface. BRDF can be represented as afunction p(θ_(h),θ_(d),ϕ_(d)) with

$\theta_{h},{\theta_{d} \in \left\lbrack {0,\frac{\pi}{2}} \right)}$

and ϕ_(d)∈[0,π). Further, upon incorporating a bivariate representationto express a per-pixel BRDF, an exemplary equation for representing theBRDF is:

${\hat{\rho}\left( {\theta_{h},\theta_{d}} \right)} = {\frac{1}{180}{\sum\limits_{\varphi_{d}}\; {\hat{\rho}\left( {\theta_{h},\theta_{d},\varphi_{d}} \right)}}}$

In such an equation, ρ is used as the bivariate representation of theBRDF. In this way, the material model can represent the BRDF of theimaged surface on a per-pixel basis. When determining materialproperties of a surface during material capture information about asurface material can be estimated using a database, such as a materiallibrary, of reference materials that have had such informationpreviously calculated. Such reference materials can be materials forwhich the BRDF was previously determined. As such, the BRDF at a pixelcan be represented using a set of reference materials selected from adictionary of reference materials. The process of narrowing thereference material library to the materials that exhibit similar surfaceproperties as the imaged surface is known as global material estimation.The selected reference materials can be represented as D=[ρ₁, ρ₂, . . .ρ_(M)], so that the BRDF at a pixel can be expressed as ρ_(p)=Dc_(p),c_(p)≥0 where c_(p) represents a coefficient of abundance of a materialthat is used to determine the properties of a surface material.

Utilizing such reference materials to determine the BRDF at a pixel,instead of solving for high-dimensional vector ρ_(p), only theabundance, or amount, of coefficient c_(p) needs to be solved for thepixel, where c_(p) is proportional to the number of reference materialsin the dictionary. For example, if a material surface is represented bya dictionary of two reference materials, the total c_(p) at a pixel addsup to 100%, for instance, c₁ has an abundance of 0.9 (90%) and c₂ has anabundance of 0.1 (10%). This means c₁ contributes 90% of materialproperties and c₂ contributes 10% of the material properties of thesurface undergoing material capture. As such, the BRDF at a pixel can berepresented as a weighted combination of the material reference BRDF.The reference materials in the database can be narrowed to a small setof materials that best represent the captured material surface. Such asubset of reference materials from the database, best representing thematerial captured in the images of the real-world surface, can be usedto determine a model that accurately captures the information about thecaptured material surface. As such, a sparse prior can be incorporatedinto the BRDF determination to restrict the number of referencematerials used and regularize such an under-constrained equation. Byassuming that the coefficient of abundance is sparse, the BRDF at apixel p becomes a linear combination of a select subset of dictionaryreference materials that best represent a surface material undergoingmaterial capture.

To fully determine the material model of the properties of the surfaceundergoing material capture, an example equation for approximating theproperties is

$\begin{matrix}{{\underset{n_{p},c_{p}}{argmin}{{I_{p} - {{B\left( {n_{p},l_{p},v_{p}} \right)} \cdot c_{p}}}}_{2}^{2}} + {\lambda {c_{p}}_{1}}} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

In such an equation, I denotes image intensities observed at pixel pafter calibration of images of the surface. l_(p) and v_(p) are thelighting and view directions determined during calibration for thecollected images so that l_(p)=[l_(p) ¹l_(p) ², . . . l_(p) ^(Q)],v_(p)=[v_(p) ¹, v_(p) ², . . . v_(p) ^(Q)]. In this way, the lightingand view direction at a pixel p as determined during calibration isrespectively denoted as l_(p) ^(i) and v_(p) ^(i) for an image I_(p)^(k).

n_(p) represents the surface normal at pixel p. Surface normal is avector that is perpendicular to a tangent plane to the surface at apoint. B can be a matrix denoted as B=D(s1^(T)), where s accounts forshading of a lighting direction. As such, s can be represented ass(i)=max (0, n_(p) ^(T)l_(p) ^(i)). As such, for each estimatedcoefficient c as described above, optimal surface normals can bedetermined when estimating material properties during material capture.

The three-dimensional positions of the pixels determined duringcalibration can be used as initial surface normals to solve forabundances using Equation 1 and a library of reference materials. Oneexample of such a library of reference materials is the MitsubishiElectric Research Laboratories (MERL) BRDF Database. The MERL BRDFDatabase contains reflectance functions of over 100 different materials,where each reflectance is stored as a densely measured BRDF determinedusing the traditional, highly time-consuming laboratory setting system.This can output a material response curve of the global materialsimitated in the surface, where the peaks in the curve correspond to thereference materials in the material library with similar properties tothe captured material surface. Upon creating a material response curve,the top materials with largest response can be selected to reduce thedictionary. The set of materials can then be used to further determinethe material properties of the captured surface. This can be donebecause the BRDF at each pixel can be determined for pixels using alinear combination of a small number of reference materials for whichthe BRDF and surface normal are already determined.

Using a set of reference materials removes the need to perform complexoptimization involving a scarcity constraint, allowing for significantspeedups and increased efficiency in the determination of the materialmodel. B can be generated using shading terms associated with the set ofreference materials. For example in Equation 1:

${\underset{n_{p},c_{p}}{argmin}{{I_{p} - {{B\left( {n_{p},l_{p},v_{p}} \right)} \cdot c_{p}}}}_{2}^{2}} + {\lambda {c_{p}}_{1}}$

the λ∥c_(p)∥₁ can be removed from the equation to become:

$\begin{matrix}{{\underset{n_{p},c_{p}}{argmin}{{I_{p} - {{B\left( {n_{p},l_{p},v_{p}} \right)} \cdot c_{p}}}}_{2}^{2}};{c_{p} > 0.}} & \left( {{Equation}\mspace{14mu} 2} \right)\end{matrix}$

To determine the material properties of the captured surface usingreference materials, iterative updates can be performed using, forexample, Equation 2, until the error in the output is reduced to below apredetermined threshold. Given the information about the referencematerials from the material dictionary, a grid based on elevation andazimuth angles can be constructed where candidate normal can beevaluated to match the per-pixel intensity profile of the imagedmaterial surface. An exemplary equation for expressing such a grid forperforming this can be represented as:

={({tilde over (θ)},{tilde over (ϕ)})∥{tilde over (θ)}−{tilde over(ϕ)}|≤

θ,|{tilde over (ϕ)}−ϕp|≤

ϕ}, where

_(Φ) and

_(ϕ) are thresholds to determine cardinality for each candidate set. Foreach element in

, a candidate surface normal can be computed as ñ=[sin {tilde over (θ)}cos {tilde over (ϕ)}, sin {tilde over (θ)} sin {tilde over (ϕ)}, cos{tilde over (θ)}]. As such, an estimate of a surface normal at pixel pcan be gives as

${\hat{n}}_{p} = {\underset{n_{p} \in}{\arg \mspace{11mu} \min}{{{I_{p} - {{\hat{B}\left( {n_{p},l_{p},v_{p}} \right)} \cdot c_{p}}}}_{2}^{2}.}}$

Such an equation can be solved by examining the candidate normal on thegrid.

Upon determining a surface normal estimate {circumflex over (n)}_(p),for example, using the above equation, an estimate of abundance, oramount, at a pixel can be determined. An equation that can be used tosolve for the abundance is

${\hat{c}}_{p} = {\arg {\; \;}{\min\limits_{c_{p}}{{I_{p} - {{\hat{B}\left( {n_{p},l_{p},v_{p}} \right)} \cdot c_{p}}}}_{2}^{2}}}$

where c_(p)≥0. Such an equation can be solved using a non-negative leastsquare. A non-negative least square is a constrained version of theleast squares problem where the coefficients are not allowed to becomenegative. Advantageously, using such a per-pixel estimation frameworkallows the system to handle complex spatial variations in the BRDF,allowing for variations in a surface to be maintained during materialcapture.

To determine the abundance of each reference material, iterative updatescan be performed using, for example, Equation 1 and/or 2, until theerror in the output is reduced to below a predetermined threshold. Toperform this determination, a subset of reference materials can beselected for a material library as described above. Given this compactgroup of reference materials, and the abundances c_(p) computed based onthe set of reference materials, the surface normal can be solved for bysearching for estimates of the surface normals that minimize the energyfunction using, for example, Equation 1 and/or 2. Upon finding newsurface normals, Equation 1 and/or 2 can then be used to solve for newabundances. This process can be iteratively performed until the outputof Equation 1 and/or 2 falls below a predetermined threshold level. Forexample, the percentage difference between two iterations falls within acertain percentage of each other, such as 1% or 0.1%.

Once the errors during the iterative updates are reduced below apredetermined threshold, the material model can be output, such as,output material model 312. Upon being output, the material modelrepresenting the surface material can be provided to a user, forexample, within an application, allowing the user to apply theattributes of the captured surface to the surface of acomputer-generated object. Such a material model can be auto appliedupon an indication that a particular surface should be applied to thesurface of a computer-generated object; for example, a user could imagethe surface of a checkered couch and indicate that the surface of thecouch should be applied to a computer-generated couch. The materialcapture system as previously described can be used to determine amaterial model for the material surface of the imaged couch andauto-apply the material model to the computer-generated couch.

FIG. 4 depicts an example configuration of an illustrative refiningengine for implementing the refining stage of determining the materialproperties of a surface material during material capture, in accordancewith various embodiments of the present disclosure. Modeling engine 402can be run on any type of processing system executed by instructions ona system run on a computing device, for example material capture system204 and/or material capture system 304. It should be understood thatthis and other arrangements described herein are set forth only asexamples. Other arrangements and components can be used in addition toor instead of those shown, and some components may be omitted altogetherfor the sake of clarity. As depicted in FIG. 4, modeling engine 402 iscomprised of dictionary reduction component 404, surface normalcomponent 406 and abundance component 408.

Once images taken of a material surface are calibrated, modeling engine402 can use the calibrated images to determine the properties of thematerial surface. Refining engine can utilize information from thecalibrated images to perform several steps on various components.

Reference material selection component 404 can be utilized to performglobal material estimation and determine the set of reference materialsto use to determine a material model of the imaged surface. Selecting aset of reference materials analyzed to determine the properties of thesurface material greatly reduces the time and computational powerrequired for material capture. To determine the set of referencematerials to utilize, three-dimensional positions of the pixelsdetermined during calibration can be used as surface normals to solvefor abundances using Equation 1 and a library of reference materials. Alibrary of reference materials can be stored, for example, in data store410. Using Equation 1 and a library of reference materials can be usedto output a material response curve where the peaks in the curvecorrespond to the materials with similar properties to the capturedmaterial surface. Upon creating a material response curve, the topmaterials with the greatest similarity (e.g., exhibiting the largestresponse peaks on the material response curve) can be selected to reducethe dictionary.

The selected set of materials can be used because the BRDF at a pixelcan be determined using a linear combination of information from thereference materials, such as a predetermined BRDF and surface normal. Assuch, surface normal component 406 can be used to estimate surfacenormals for the surface material using information about the referencematerials from the material library. For example, if the subset ofmaterials reduces a library from over 100 reference materials down totwo reference materials, information about those two reference materialscan be used to determine an estimated surface normal that represents thesurface material undergoing material capture.

Upon determining surface normals, the BRDF can then be estimated usingabundance component 408 to determine abundance levels, or amounts. forthe BRDF from the reference materials. For example, if a materialsurface is represented by a subset of two reference materials from amaterial library, the total abundance at a pixel will add up to 100%,for instance, if c₁ has an abundance of 0.6 (90%) then c₂ will have anabundance of 0.4 (10%). As such, the abundance at a pixel can berepresented as a weighted combination of the material reference BRDF.

Surface normal component 406 and abundance component 408 can be used toiteratively update the material model until the accuracy is within apredetermined threshold, for instance, indicated by a percentage changebetween two iterations below a certain amount, for example 0.1%. Suchaccuracy can be determined using the difference between the energyfunction determined using, for example, Equation 2 over two or moreiterations.

In embodiments, modeling engine 402 can store the optimized model of thesurface material properties. The optimized model can be stored in datastore 410. Data store 410 can be updated by modeling engine 402 when anoptimized material model is determined to be within a thresholdpercentage of error between iterative updates to determine an optimalrepresentation of the captured material. Having such a data store allowsfor material models of various materials to be stored. In addition, sucha data store can be used to store a library of reference materials usedto determine the properties of a surface material during materialcapture.

FIG. 5 illustrates a process flow 500 depicting an example determinationof material properties during material capture of a surface, inaccordance with various embodiments of the present disclosure. Processflow 500 can be carried out, for example, by material capture system asdiscussed with reference to FIG. 3.

As depicted, process flow 500 begins at block 502 where images of thematerial surface are received. These images can be captured using acommodity device, for instance, a mobile phone. The captured surfaceshould be largely planar, however, small surface variations can beaccounted for during the material capture process, for example, whereone portion of the surface is 1 cm away from the capture device andanother portion of the surface is 1.01 cm away from the capture device.Such images can be captured while a light source on the device isilluminated, providing changing lighting on the surface beingcalibrated.

At block 504, the received images can be calibrated. The received imagescan be calibrated by aligning pixels in the images and then determiningthe different lighting and/or camera positions at corresponding pixelsacross the images. In particular, the received images can be alignedutilizing, for example, a grid with a point that is constant across allcaptured images.

At block 506, a set of reference materials can be selected from adictionary of reference materials, where the set most closely reflect tothe properties of the captured material surface. This subset ofreference materials can be used to determine the material properties ofthe captured material surface. To determine such a subset, abundancecoefficients can be calculated with sparse priors usingthree-dimensional information of pixels determined during calibration ofthe images. These coefficients are then summed across pixels. This canbe used to produce a material response curve where the peaks in thecurve correspond to the materials with similar properties to thecaptured material surface. Upon creating the material response curve,the top materials with largest response can be selected to reduce thedictionary. Using such a subset of reference materials removes the needto perform complex optimization involving a scarcity constraint,allowing for significant speedups in the determination of the materialmodel. For example, Equation 1 can be reduced to Equation 2. Such asubset of materials can be used because the BRDF at each pixel can bedetermined for pixels using a linear combination of a small number ofreference materials for which the BRDF and surface normal are alreadydetermined.

Once the dictionary of reference materials is reduced to the subsetusing the response curve, the process then moves to block 508 wheresurface normals for the material can be solved for using, for exampleEquations 2. To determine the surface normals, a grid based on elevationand azimuth angles can be specified and searched for candidate normalsthat best match the profile of the pixels of the captured surface. Atblock 510, the surface normals determined at block 508 can be used todetermine the BRDF of pixels using abundance of the predetermined BRDFof the subset of reference materials.

After determining the abundance, the process can move to block 512 wherea determination is made as to whether error of the material model hasbeen reduced below a predetermined threshold. If there is not an error,then the process repeats blocks 508-510 as described above. If there isan error, then the process proceeds to block 514. At bock 514, thematerial model is output. Such a material model representing the surfacematerial can be provided to a user, for example, within an application,allowing the user to apply the attributes of the captures surface to thesurface of a computer-generated object.

FIG. 6 illustrates a process flow 600 depicting an example calibrationof images used for material capture, in accordance with variousembodiments of the present disclosure. Such calibration can take placeusing a calibration engine of a material capture system as describedwith reference to FIG. 3.

At block 602, captured images are received. Such images can be capturedusing, for example, a user device such as user device 202 a-202 n asdescribed with reference to FIG. 2. The captured images can be taken ofa largely planar surface with minor variations. Additionally, the imagescan be taken with an illuminated light source on the capturing device,for example, a mobile phone with an illuminated flash. It should beappreciated that such images can include a set of images and/or asequence of video.

At block 604, the rotation and translation of the position of the cameraused to capture the images received at block 602 can be determined. Inaddition, at block 604, the position of light source in relation to thecamera can be determined. This is possible because the light source andthe camera have fixed positions in relation to one another, as such, itis possible to acquire the position of the light source in the capturedimages. A rotation and translation matrix can also be determined foreach camera position.

At block 606, the pixels of the captured images can be aligned. Thisalignment can be carried out using a grid. A checkerboard can be placedinto the frame when a surface material is being imaged so that thecaptured images contain images of the checkerboard and the materialsurface. Having a checkerboard that is constant in the images allows foraligning the images using specified portions of the checkerboard tocalibrate the images. In this way, a checkerboard allows fordetermination of the rotation and translation matrix of the cameraposition.

As such, once the pixels are aligned and the rotation and translationmatrix for the camera positions and the position of the light source aredetermined, at block 608, the lighting and view direction of pixels canbe determined. Such information about the lighting and view direction atpixels can be used, for example, in Equations 1 and 2 to determine theproperties of the material surface undergoing material capture.

Finally, the three-dimensional position of pixels can be determined atblock 610. This information can be used, for example, in Equation 1 toperform global material estimation and reduce Equation 1 to Equation 2by determining a subset of reference materials that best represent thesurface undergoing material capture.

Having described embodiments of the present invention, an exampleoperating environment in which embodiments of the present invention maybe implemented is described below in order to provide a general contextfor various aspects of the present invention. Referring to FIG. 7, anillustrative operating environment for implementing embodiments of thepresent invention is shown and designated generally as computing device700. Computing device 700 is but one example of a suitable computingenvironment and is not intended to suggest any limitation as to thescope of use or functionality of the invention. Neither should thecomputing device 700 be interpreted as having any dependency orrequirement relating to any one or combination of componentsillustrated.

Embodiments of the invention may be described in the general context ofcomputer code or machine-useable instructions, includingcomputer-executable instructions such as program modules, being executedby a computer or other machine, such as a smartphone or other handhelddevice. Generally, program modules, or engines, including routines,programs, objects, components, data structures, etc., refer to code thatperform particular tasks or implement particular abstract data types.Embodiments of the invention may be practiced in a variety of systemconfigurations, including hand-held devices, consumer electronics,general-purpose computers, more specialized computing devices, etc.Embodiments of the invention may also be practiced in distributedcomputing environments where tasks are performed by remote-processingdevices that are linked through a communications network.

With reference to FIG. 7, computing device 700 includes a bus 710 thatdirectly or indirectly couples the following devices: memory 712, one ormore processors 714, one or more presentation components 716,input/output ports 718, input/output components 720, and an illustrativepower supply 722. Bus 710 represents what may be one or more busses(such as an address bus, data bus, or combination thereof). Although thevarious blocks of FIG. 7 are shown with clearly delineated lines for thesake of clarity, in reality, such delineations are not so clear andthese lines may overlap. For example, one may consider a presentationcomponent such as a display device to be an I/O component, as well.Also, processors generally have memory in the form of cache. Werecognize that such is the nature of the art, and reiterate that thediagram of FIG. 7 is merely illustrative of an example computing devicethat can be used in connection with one or more embodiments of thepresent disclosure. Distinction is not made between such categories as“workstation,” “server,” “laptop,” “hand-held device,” etc., as all arecontemplated within the scope of FIG. 7 and reference to “computingdevice.”

Computing device 700 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 700 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media.

Computer storage media include volatile and nonvolatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer-readable instructions, data structures,program modules or other data. Computer storage media includes, but isnot limited to, RAM, ROM, EEPROM, flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other optical diskstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or any other medium which can be used tostore the desired information and which can be accessed by computingdevice 700. Computer storage media excludes signals per se.

Communication media typically embodies computer-readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and other wireless media. Combinations of any ofthe above should also be included within the scope of computer-readablemedia.

Memory 712 includes computer storage media in the form of volatileand/or nonvolatile memory. As depicted, memory 712 includes instructions724. Instructions 724, when executed by processor(s) 714 are configuredto cause the computing device to perform any of the operations describedherein, in reference to the above discussed figures, or to implement anyprogram modules described herein. The memory may be removable,non-removable, or a combination thereof. Illustrative hardware devicesinclude solid-state memory, hard drives, optical-disc drives, etc.Computing device 700 includes one or more processors that read data fromvarious entities such as memory 712 or I/O components 720. Presentationcomponent(s) 716 present data indications to a user or other device.Illustrative presentation components include a display device, speaker,printing component, vibrating component, etc.

I/O ports 718 allow computing device 700 to be logically coupled toother devices including I/O components 720, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, etc.

Embodiments presented herein have been described in relation toparticular embodiments which are intended in all respects to beillustrative rather than restrictive. Alternative embodiments willbecome apparent to those of ordinary skill in the art to which thepresent disclosure pertains without departing from its scope.

From the foregoing, it will be seen that this disclosure in one welladapted to attain all the ends and objects hereinabove set forthtogether with other advantages which are obvious and which are inherentto the structure.

It will be understood that certain features and sub-combinations are ofutility and may be employed without reference to other features orsub-combinations. This is contemplated by and is within the scope of theclaims.

In the preceding detailed description, reference is made to theaccompanying drawings which form a part hereof wherein like numeralsdesignate like parts throughout, and in which is shown, by way ofillustration, embodiments that may be practiced. It is to be understoodthat other embodiments may be utilized and structural or logical changesmay be made without departing from the scope of the present disclosure.Therefore, the preceding detailed description is not to be taken in alimiting sense, and the scope of embodiments is defined by the appendedclaims and their equivalents.

Various aspects of the illustrative embodiments have been describedusing terms commonly employed by those skilled in the art to convey thesubstance of their work to others skilled in the art. However, it willbe apparent to those skilled in the art that alternate embodiments maybe practiced with only some of the described aspects. For purposes ofexplanation, specific numbers, materials, and configurations are setforth in order to provide a thorough understanding of the illustrativeembodiments. However, it will be apparent to one skilled in the art thatalternate embodiments may be practiced without the specific details. Inother instances, well-known features have been omitted or simplified inorder not to obscure the illustrative embodiments.

Various operations have been described as multiple discrete operations,in turn, in a manner that is most helpful in understanding theillustrative embodiments; however, the order of description should notbe construed as to imply that these operations are necessarily orderdependent. In particular, these operations need not be performed in theorder of presentation. Further, descriptions of operations as separateoperations should not be construed as requiring that the operations benecessarily performed independently and/or by separate entities.Descriptions of entities and/or modules as separate modules shouldlikewise not be construed as requiring that the modules be separateand/or perform separate operations. In various embodiments, illustratedand/or described operations, entities, data, and/or modules may bemerged, broken into further sub-parts, and/or omitted.

The phrase “in one embodiment” or “in an embodiment” is used repeatedly.The phrase generally does not refer to the same embodiment; however, itmay. The terms “comprising,” “having,” and “including”” are synonymous,unless the context dictates otherwise. The phrase “A/B” means “A or B.”The phrase “A and/or B” means “(A), (B), or (A and B).” The phrase “atleast one of A, B and C” means “(A), (B), (C), (A and B), (A and C), (Band C) or (A, B and C).”

What is claimed is:
 1. A computer-implemented method, comprising:determining reflectance information for a surface depicted in aplurality of images; selecting a set of reference materialsrepresentative of properties of the surface, wherein the set ofreference materials have known material properties; and generating amaterial model of the surface using the set of reference materials. 2.The computer-implemented method claim 1, further comprising: aligningthe plurality of images based on alignment of corresponding pixelsassociated with each of the plurality of images.
 3. Thecomputer-implemented method claim 1, wherein determining the reflectanceinformation includes finding lighting and view directions for at leastone pixel of the plurality of images.
 4. The computer-implemented methodof claim 1, further comprising: determining a surface normal estimate ofa pixel of corresponding pixels from the plurality of images;determining a material coefficient abundance estimate at the pixel;iteratively updating the surface normal estimate and the materialcoefficient abundance estimate at the pixel until error is reduced belowa predetermined threshold.
 5. The computer-implemented method of claim4, wherein the surface normal estimate is determined utilizing a localneighborhood search of a grid based on elevation and azimuth angles forthe set of reference materials.
 6. The computer-implemented method ofclaim 4, wherein the material coefficient abundance estimate isdetermined utilizing a non-negative least squares approach based oninputting the surface normal estimate.
 7. The computer-implementedmethod of claim 1, further comprising: determining a surface normalestimate at a pixel of corresponding pixels from the plurality ofimages; determining a material coefficient abundance estimate at thepixel; updating the surface normal estimate and updating the materialcoefficient abundance estimate at the pixel in parallel until error isreduced below a predetermined threshold.
 8. The computer-implementedmethod of claim 1, wherein selecting the set of reference materialsincludes generating a sparse response curve for a library of referencematerials, wherein reference materials with a response on the sparseresponse curve are selected as the set of reference materials.
 9. Thecomputer-implemented method of claim 1, wherein a user can apply thematerial model of the material surface to a surface of acomputer-generated object during rendering of the computer-generatedobject.
 10. One or more non-transitory computer-readable storage mediahaving instructions stored thereon, which, when executed by one or moreprocessors of a computing device, cause the computing device to:determine reflectance information for a surface depicted in a pluralityof images; select a set of reference materials representative ofproperties of the surface, wherein the set of reference materials haveknown material properties; and generate a material model of the surfaceusing the set of reference materials.
 11. The one or more non-transitorycomputer-readable storage media of claim 10, further cause the computingdevice to: calibrate the plurality of images, wherein calibrationutilizes a grid depicted in the plurality of images to align theplurality of images and determine reflectance information for at least aportion of the aligned plurality of images; identify the set ofreference materials using the calibrated plurality of images; determinematerial properties for the material surface at a selected pixel usingthe set of reference materials; and output a material model of thematerial surface based on the determined material properties.
 12. Theone or more non-transitory computer-readable storage media of claim 10,wherein the reflectance information includes lighting and viewdirections for at least one pixel of the plurality of images.
 13. Theone or more non-transitory computer-readable storage media of claim 10,wherein instructions further cause the computing device to: determine asurface normal estimate at a pixel of corresponding pixels from theplurality of images; determine a material coefficient abundance estimateat the pixel; iteratively update the surface normal estimate and updatethe material coefficient abundance estimate at the pixel until error isreduced below a predetermined threshold.
 14. The one or morenon-transitory computer-readable storage media of claim 12, wherein thesurface normal estimate is determined utilizing a local neighborhoodsearch of a grid based on elevation and azimuth angles for the set ofreference materials.
 15. The one or more non-transitorycomputer-readable storage media of claim 12, wherein the materialcoefficient abundance estimate is determined utilizing a non-negativeleast squares approach based on inputting the surface normal estimate.16. The one or more non-transitory computer-readable storage media ofclaim 10, wherein instructions further cause the computing device to:determine a surface normal estimate at a pixel of corresponding pixelsfrom the plurality of images; determine a material coefficient abundanceestimate at the pixel; update the surface normal estimate and update thematerial coefficient abundance estimate at the pixel in parallel untilerror is reduced below a predetermined threshold.
 17. The one or morenon-transitory computer-readable storage media of claim 10, whereindetermining global material estimation includes generating a sparseresponse curve for a library of reference materials, wherein referencematerials with a response on the sparse response curve are selected asthe set of reference materials.
 18. A computing system comprising: meansfor determining reflectance information for a surface depicted in aplurality of images; means for selecting a set of reference materialsrepresentative of properties of the surface, wherein the set ofreference materials have known material properties; and means forgenerating a material model of the surface using the set of referencematerials.
 19. The computing system of claim 18 further comprising:means for calibrating the plurality of images; and means for determiningmaterial properties for the material surface at a selected pixel usingthe set of reference materials.
 20. The computing system of claim 19,further comprising means for determining a surface normal estimate at aselected pixel; means for determining a material coefficient abundanceestimate at the selected pixel; means for determining a materialcoefficient abundance estimate at the selected pixel; means for updatingthe surface normal estimate and updating the material coefficientabundance estimate at the selected pixel until error is reduced below apredetermined threshold.